System for identifying handwritten characters on an image using a classification model

ABSTRACT

A system for identifying handwritten characters on an image using a classification model that employs a neural network. The system includes a computer having a processor and a memory device that stores data and executable code that, when executed, causes the processor to read and convert typed text on the image to machine encoded text to identify locations of the typed text on the image; identify a location on the image that includes handwritten text based on the location of predetermined typed text on the image; identify clusters of non-white pixels in the image at the location having the handwritten text, where constraints are employed to refine and limit the clusters; generate an individual and separate cluster image for each identified cluster; and classify each cluster image using machine learning and at least one neural network to determine the likelihood that the cluster is a certain character.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application tracing priority toco-pending U.S. application Ser. No. 17/661,654 filed on May 2, 2022,the entirety of which is herein expressly incorporated by reference.

BACKGROUND Field

This disclosure relates generally to a system and method for identifyinghandwritten characters on an image and, more particularly, to a systemand method for identifying handwritten characters on an image using aclassification model that employs a neural network, where the system andmethod identify clusters of non-white pixels in the image at a locationhaving handwritten text, and where each cluster is presumed to be ahandwritten character, and where the processor employs constraints sothat separate clusters are not identified in a stacked up/down directionrelative to a writing direction of the handwritten text, that limits asize of each cluster to be less than a predetermined size and/or thatrequires all of the clusters to be within a certain percentage size ofeach other.

Discussion

A bank is a financial institution that is licensed to receive depositsfrom individuals and organizations and to make loans to thoseindividuals and organizations or others. Banks may also perform otherservices such as wealth management, currency exchange, etc. Therefore, abank may have thousands of customers and clients. Depending on theservices that a bank provides, it may be classified as a retail bank, acommercial bank, an investment bank or some combination thereof. Aretail bank typically provides services such as checking and savingsaccounts, loan and mortgage services, financing for automobiles, andshort-term loans such as overdraft protection. A commercial banktypically provides credit services, cash management, commercial realestate services, employer services, trade finance, etc. An investmentbank typically provides corporate clients with complex services andfinancial transactions such as underwriting and assisting with mergerand acquisition activity.

Most people and businesses have a checking account that they writechecks from to pay other people and businesses for goods and services.When those other people and businesses deposit the check into theiraccount the check is generally scanned as an image and submitted to anautomated clearing house (ACH) to process the transaction, which is acomputer-based electronic network for processing transactions. Throughthis process the image of the check is returned to the bank from whichit was drawn to be paid and is automatically connected to the accountthat it was drawn from. The transaction record and check image isaccessible and displayed for the owner of the account online and througha mobile application on a transaction register. However, for handwrittenchecks the automated process that populates the transaction registergenerally only shows the transaction date, the check number and theamount because the handwritten payee is not detectable by currentautomated processes. If the owner wants to see the payee of the check,he needs to click on the check image, which is a layer of inconvenience.

SUMMARY

The following discussion discloses and describes a system and method foridentifying handwritten characters on an image using a classificationmodel that employs a neural network. The system includes a computerhaving a processor and a memory device that stores data and executablecode that, when executed, causes the processor to read and convert typedtext on the image to machine encoded text to identify locations of thetyped text on the image; identify a location on the image that includeshandwritten text based on the location of predetermined typed text onthe image; identify clusters of non-white pixels in the image at thelocation having the handwritten text, where each cluster is presumed tobe a handwritten character, and where the processor employs constraintsso that separate clusters are not identified in a stacked up/downdirection relative to a writing direction of the handwritten text, thatlimits a size of each cluster to be less than a predetermined sizeand/or that requires all of the clusters to be within a certainpercentage size of each other; generate an individual and separatecluster image for each identified cluster; classify each cluster imageusing machine learning and at least one neural network to determine thelikelihood that the cluster is a certain character; and determine whatcharacter each cluster image is based on the classification.

Additional features of the disclosure will become apparent from thefollowing description and appended claims, taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system and environment thereof by which a userbenefits through use of services and products of an enterprise system;

FIG. 2 is a diagram of a feedforward network;

FIG. 3 is a diagram of a convolutional neural network (CNN);

FIG. 4 is a diagram of a portion of the CNN shown in FIG. 3 illustratingassigned weights at connections or neurons;

FIG. 5 is a diagram representing an exemplary weighted sum computationin a node in an artificial neural network;

FIG. 6 is a diagram of a recurrent neural network (RNN) utilized inmachine learning;

FIG. 7 is a schematic logic diagram of an artificial intelligenceprocessor operating an artificial intelligence program;

FIG. 8 is a flow chart showing a method for model development anddeployment by machine learning;

FIG. 9 is a flow diagram illustrating a process for identifyinghandwritten characters on an image;

FIG. 10 is a representation of a 28×28 pixel cluster image that isgenerated by the process shown in FIG. 9 ; and

FIG. 11 is an illustration of a neural network that can be used in acharacter classification model in the process shown in FIG. 9 .

DETAILED DESCRIPTION OF THE EMBODIMENTS

The following discussion of the embodiments of the disclosure directedto a system and method for identifying handwritten characters on animage using a classification model that employs a neural network, wherethe system and method identify clusters of non-white pixels in the imageat a location having handwritten text, and where each cluster ispresumed to be a handwritten character, and where the processor employsconstraints so that separate clusters are not identified in a stackedup/down direction relative to a writing direction of the handwrittentext, that limits a size of each cluster to be less than a predeterminedsize and/or that requires all of the clusters to be within a certainpercentage size of each other is merely exemplary in nature, and is inno way intended to limit the disclosure or its applications or uses.

Embodiments of the present disclosure will now be described more fullyhereinafter with reference to the accompanying drawings, in which some,but not all, embodiments of the disclosure are shown. Indeed, thedisclosure may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. Like numbers refer to like elements throughout.Unless described or implied as exclusive alternatives, featuresthroughout the drawings and descriptions should be taken as cumulative,such that features expressly associated with some particular embodimentscan be combined with other embodiments. Unless defined otherwise,technical and scientific terms used herein have the same meaning ascommonly understood to one of ordinary skill in the art to which thepresently disclosed subject matter pertains.

The exemplary embodiments are provided so that this disclosure will beboth thorough and complete, and will fully convey the scope of thedisclosure and enable one of ordinary skill in the art to make, use andpractice the disclosure.

The terms “coupled,” “fixed,” “attached to,” “communicatively coupledto,” “operatively coupled to,” and the like refer to both (i) directconnecting, coupling, fixing, attaching, communicatively coupling; and(ii) indirect connecting coupling, fixing, attaching, communicativelycoupling via one or more intermediate components or features, unlessotherwise specified herein. “Communicatively coupled to” and“operatively coupled to” can refer to physically and/or electricallyrelated components.

Embodiments of the present disclosure described herein, with referenceto flowchart illustrations and/or block diagrams of methods orapparatuses (the term “apparatus” includes systems and computer programproducts), will be understood such that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a particular machine, such that the instructions, which executevia the processor of the computer or other programmable data processingapparatus, create mechanisms for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer readablememory produce an article of manufacture including instructions, whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions, which execute on the computer or other programmableapparatus, provide steps for implementing the functions/acts specifiedin the flowchart and/or block diagram block or blocks. Alternatively,computer program implemented steps or acts may be combined with operatoror human implemented steps or acts in order to carry out an embodimentof the disclosure.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of, and not restrictive on, the broad disclosure,and that this disclosure not be limited to the specific constructionsand arrangements shown and described, since various other changes,combinations, omissions, modifications and substitutions, in addition tothose set forth in the above paragraphs, are possible. Those skilled inthe art will appreciate that various adaptations, modifications, andcombinations of the herein described embodiments can be configuredwithout departing from the scope and spirit of the disclosure.Therefore, it is to be understood that, within the scope of the includedclaims, the disclosure may be practiced other than as specificallydescribed herein.

FIG. 1 illustrates a system 10, such as a banking system, andenvironment thereof by which a user 18 benefits through use of servicesand products of an enterprise system 12. The user 18 accesses servicesand products by use of one or more user devices, illustrated in separateexamples as a computing device 14 and a mobile device 16, which may be,as non-limiting examples, a smart phone, a portable digital assistant(PDA), a pager, a mobile television, a gaming device, a laptop computer,a camera, a video recorder, an audio/video player, radio, a GPS device,or any combination of the aforementioned, or other portable device withprocessing and communication capabilities. In the illustrated example,the mobile device 16 is the system 10 as having exemplary elements, thebelow descriptions of which apply as well to the computing device 14,which can be, as non-limiting examples, a desktop computer, a laptopcomputer or other user-accessible computing device.

Furthermore, the user device, referring to either or both of thecomputing device 14 and the mobile device 16, may be or include aworkstation, a server, or any other suitable device, including a set ofservers, a cloud-based application or system, or any other suitablesystem, adapted to execute, for example any suitable operating system,including Linux, UNIX, Windows, macOS, iOS, Android and any other knownoperating system used on personal computers, central computing systems,phones, and other devices.

The user 18 can be an individual, a group, or any entity in possessionof or having access to the user device, referring to either or both ofthe computing device 14 and the mobile device 16, which may be personalor public items. Although the user 18 may be singly represented in somedrawings, at least in some embodiments according to these descriptionsthe user 18 is one of many such that a market or community of users,consumers, customers, business entities, government entities, clubs, andgroups of any size are all within the scope of these descriptions.

The user device, as illustrated with reference to the mobile device 16,includes components such as at least one of each of a processing device20, and a memory device 22 for processing use, such as random accessmemory (RAM), and read-only memory (ROM). The illustrated mobile device16 further includes a storage device 24 including at least one of anon-transitory storage medium, such as a microdrive, for long-term,intermediate-term, and short-term storage of computer-readableinstructions 26 for execution by the processing device 20. For example,the instructions 26 can include instructions for an operating system andvarious applications or programs 30, of which the application 32 isrepresented as a particular example. The storage device 24 can storevarious other data items 34, which can include, as non-limitingexamples, cached data, user files such as those for pictures, audioand/or video recordings, files downloaded or received from otherdevices, and other data items preferred by the user or required orrelated to any or all of the applications or programs 30.

The memory device 22 is operatively coupled to the processing device 20.As used herein, memory includes any computer readable medium to storedata, code, or other information. The memory device 22 may includevolatile memory, such as volatile RAM including a cache area for thetemporary storage of data. The memory device 22 may also includenon-volatile memory, which can be embedded and/or may be removable. Thenon-volatile memory can additionally or alternatively include anelectrically erasable programmable read-only memory (EEPROM), flashmemory or the like.

The memory device 22 and the storage device 24 can store any of a numberof applications that comprise computer-executable instructions and codeexecuted by the processing device 20 to implement the functions of themobile device 16 described herein. For example, the memory device 22 mayinclude such applications as a conventional web browser applicationand/or a mobile P2P payment system client application. Theseapplications also typically provide a graphical user interface (GUI) ona display 40 that allows the user 18 to communicate with the mobiledevice 16, and, for example, a mobile banking system, and/or otherdevices or systems. In one embodiment, when the user 18 decides toenroll in a mobile banking program, the user 18 downloads or otherwiseobtains the mobile banking system client application from a mobilebanking system, for example, the enterprise system 12, or from adistinct application server. In other embodiments, the user 18 interactswith a mobile banking system via a web browser application in additionto, or instead of, the mobile P2P payment system client application.

The processing device 20, and other processors described herein,generally include circuitry for implementing communication and/or logicfunctions of the mobile device 16. For example, the processing device 20may include a digital signal processor, a microprocessor, and variousanalog to digital converters, digital to analog converters, and/or othersupport circuits. Control and signal processing functions of the mobiledevice 16 are allocated between these devices according to theirrespective capabilities. The processing device 20 thus may also includethe functionality to encode and interleave messages and data prior tomodulation and transmission. The processing device 20 can additionallyinclude an internal data modem. Further, the processing device 20 mayinclude functionality to operate one or more software programs, whichmay be stored in the memory device 22, or in the storage device 24. Forexample, the processing device 20 may be capable of operating aconnectivity program, such as a web browser application. The web browserapplication may then allow the mobile device 16 to transmit and receiveweb content, such as, for example, location-based content and/or otherweb page content, according to a wireless application protocol (WAP),hypertext transfer protocol (HTTP), and/or the like.

The memory device 22 and the storage device 24 can each also store anyof a number of pieces of information, and data, used by the user deviceand the applications and devices that facilitate functions of the userdevice, or are in communication with the user device, to implement thefunctions described herein and others not expressly described. Forexample, the storage device 24 may include such data as userauthentication information, etc.

The processing device 20, in various examples, can operatively performcalculations, can process instructions for execution and can manipulateinformation. The processing device 20 can execute machine-executableinstructions stored in the storage device 24 and/or the memory device 22to thereby perform methods and functions as described or implied herein,for example, by one or more corresponding flow charts expressly providedor implied as would be understood by one of ordinary skill in the art towhich the subject matters of these descriptions pertain. The processingdevice 20 can be or can include, as non-limiting examples, a centralprocessing unit (CPU), a microprocessor, a graphics processing unit(GPU), a microcontroller, an application-specific integrated circuit(ASIC), a programmable logic device (PLD), a digital signal processor(DSP), a field programmable gate array (FPGA), a state machine, acontroller, gated or transistor logic, discrete physical hardwarecomponents, and combinations thereof. In some embodiments, particularportions or steps of methods and functions described herein areperformed in whole or in part by way of the processing device 20, whilein other embodiments methods and functions described herein includecloud-based computing in whole or in part such that the processingdevice 20 facilitates local operations including, as non-limitingexamples, communication, data transfer, and user inputs and outputs suchas receiving commands from and providing displays to the user.

The mobile device 16, as illustrated, includes an input and outputsystem 36, referring to, including, or operatively coupled with, userinput devices and user output devices, which are operatively coupled tothe processing device 20. The user output devices include the display 40(e.g., a liquid crystal display or the like), which can be, as anon-limiting example, a touch screen of the mobile device 16, whichserves both as an output device, by providing graphical and text indiciaand presentations for viewing by one or more of the users 18, and as aninput device, by providing virtual buttons, selectable options, avirtual keyboard, and other indicia that, when touched, control themobile device 16 by user action. The user output devices include aspeaker 44 or other audio device. The user input devices, which allowthe mobile device 16 to receive data and actions such as buttonmanipulations and touches from a user such as the user 18, may includeany of a number of devices allowing the mobile device 16 to receive datafrom a user, such as a keypad, keyboard, touch-screen, touchpad,microphone 42, mouse, joystick, other pointer device, button, soft key,and/or other input device(s). The user interface may also include acamera 46, such as a digital camera.

Further non-limiting examples include one or more of each, any and allof a wireless or wired keyboard, a mouse, a touchpad, a button, aswitch, a light, an LED, a buzzer, a bell, a printer and/or other userinput devices and output devices for use by or communication with theuser 18 in accessing, using, and controlling, in whole or in part, theuser device, referring to either or both of the computing device 14 andthe mobile device 16. Inputs by one or more of the users 18 can thus bemade via voice, text or graphical indicia selections. For example, suchinputs in some examples correspond to user-side actions andcommunications seeking services and products of the enterprise system12, and at least some outputs in such examples correspond to datarepresenting enterprise-side actions and communications in two-waycommunications between the user 18 and the enterprise system 12.

The mobile device 16 may also include a positioning system device 48,which can be, for example, a global positioning system (GPS) deviceconfigured to be used by a positioning system to determine a location ofthe mobile device 16. For example, the positioning system device 48 mayinclude a GPS transceiver. In some embodiments, the positioning systemdevice 48 includes an antenna, transmitter, and receiver. For example,in one embodiment, triangulation of cellular signals may be used toidentify the approximate location of the mobile device 16. In otherembodiments, the positioning device 48 includes a proximity sensor ortransmitter, such as an RFID tag, that can sense or be sensed by devicesknown to be located proximate a merchant or other location to determinethat the consumer mobile device 16 is located proximate these knowndevices.

In the illustrated example, a system intraconnect 38, connects, forexample electrically, the various described, illustrated, and impliedcomponents of the mobile device 16. The intraconnect 38, in variousnon-limiting examples, can include or represent, a system bus, ahigh-speed interface connecting the processing device 20 to the memorydevice 22, individual electrical connections among the components, andelectrical conductive traces on a motherboard common to some or all ofthe above-described components of the user device. As discussed herein,the system intraconnect 38 may operatively couple various componentswith one another, or in other words, electrically connects thosecomponents, either directly or indirectly—by way of intermediatecomponent(s)—with one another.

The user device, referring to either or both of the computing device 14and the mobile device 16, with particular reference to the mobile device16 for illustration purposes, includes a communication interface 50, bywhich the mobile device 16 communicates and conducts transactions withother devices and systems. The communication interface 50 may includedigital signal processing circuitry and may provide two-waycommunications and data exchanges, for example, wirelessly via wirelesscommunication device 52, and for an additional or alternative example,via wired or docked communication by mechanical electrically conductiveconnector 54. Communications may be conducted via various modes orprotocols, of which GSM voice calls, SMS, EMS, MMS messaging, TDMA,CDMA, PDC, WCDMA, CDMA2000, and GPRS, are all non-limiting andnon-exclusive examples. Thus, communications can be conducted, forexample, via the wireless communication device 52, which can be orinclude a radio-frequency transceiver, a Bluetooth device, Wi-Fi device,a near-field communication device, and other transceivers. In addition,GPS may be included for navigation and location-related data exchanges,ingoing and/or outgoing. Communications may also or alternatively beconducted via the connector 54 for wired connections such by USB,Ethernet, and other physically connected modes of data transfer.

The processing device 20 is configured to use the communicationinterface 50 as, for example, a network interface to communicate withone or more other devices on a network. In this regard, thecommunication interface 50 utilizes the wireless communication device 52as an antenna operatively coupled to a transmitter and a receiver(together a “transceiver”) included with the communication interface 50.The processing device 20 is configured to provide signals to and receivesignals from the transmitter and receiver, respectively. The signals mayinclude signaling information in accordance with the air interfacestandard of the applicable cellular system of a wireless telephonenetwork. In this regard, the mobile device 16 may be configured tooperate with one or more air interface standards, communicationprotocols, modulation types, and access types. By way of illustration,the mobile device 16 may be configured to operate in accordance with anyof a number of first, second, third, fourth or fifth-generationcommunication protocols and/or the like. For example, the mobile device16 may be configured to operate in accordance with second-generation(2G) wireless communication protocols IS-136 (time division multipleaccess (TDMA)), GSM (global system for mobile communication), and/orIS-95 (code division multiple access (CDMA)), or with third-generation(3G) wireless communication protocols, such as universal mobiletelecommunications System (UMTS), CDMA2000, wideband CDMA (WCDMA) and/ortime division-synchronous CDMA (TD-SCDMA), with fourth-generation (4G)wireless communication protocols such as long-term evolution (LTE),fifth-generation (5G) wireless communication protocols, Bluetooth lowenergy (BLE) communication protocols such as Bluetooth 5.0,ultra-wideband (UWB) communication protocols, and/or the like. Themobile device 16 may also be configured to operate in accordance withnon-cellular communication mechanisms, such as via a wireless local areanetwork (WLAN) or other communication/data networks.

The communication interface 50 may also include a payment networkinterface. The payment network interface may include software, such asencryption software, and hardware, such as a modem, for communicatinginformation to and/or from one or more devices on a network. Forexample, the mobile device 16 may be configured so that it can be usedas a credit or debit card by, for example, wirelessly communicatingaccount numbers or other authentication information to a terminal of thenetwork. Such communication could be performed via transmission over awireless communication protocol such as the Near-field communicationprotocol.

The mobile device 16 further includes a power source 28, such as abattery, for powering various circuits and other devices that are usedto operate the mobile device 16. Embodiments of the mobile device 16 mayalso include a clock or other timer configured to determine and, in somecases, communicate actual or relative time to the processing device 20or one or more other devices. For a further example, the clock mayfacilitate timestamping transmissions, receptions, and other data forsecurity, authentication, logging, polling, data expiry and forensicpurposes.

The system 10 as illustrated diagrammatically represents at least oneexample of a possible implementation, where alternatives, additions, andmodifications are possible for performing some or all of the describedmethods, operations and functions. Although shown separately, in someembodiments, two or more systems, servers, or illustrated components mayutilized. In some implementations, the functions of one or more systems,servers, or illustrated components may be provided by a single system orserver. In some embodiments, the functions of one illustrated system orserver may be provided by multiple systems, servers, or computingdevices, including those physically located at a central facility, thoselogically local, and those located as remote with respect to each other.

The enterprise system 12 can offer any number or type of services andproducts to one or more of the users 18. In some examples, theenterprise system 12 offers products, and in some examples, theenterprise system 12 offers services. Use of “service(s)” or“product(s)” thus relates to either or both in these descriptions. Withregard, for example, to online information and financial services,“service” and “product” are sometimes termed interchangeably. Innon-limiting examples, services and products include retail services andproducts, information services and products, custom services andproducts, predefined or pre-offered services and products, consultingservices and products, advising services and products, forecastingservices and products, internet products and services, social media, andfinancial services and products, which may include, in non-limitingexamples, services and products relating to banking, checking, savings,investments, credit cards, automatic-teller machines, debit cards,loans, mortgages, personal accounts, business accounts, accountmanagement, credit reporting, credit requests and credit scores.

To provide access to, or information regarding, some or all the servicesand products of the enterprise system 12, automated assistance may beprovided by the enterprise system 12. For example, automated access touser accounts and replies to inquiries may be provided byenterprise-side automated voice, text, and graphical displaycommunications and interactions. In at least some examples, any numberof human agents 60, can be employed, utilized, authorized or referred bythe enterprise system 12. Such human agents 60 can be, as non-limitingexamples, point of sale or point of service (POS) representatives,online customer service assistants available to the users 18, advisors,managers, sales team members, and referral agents ready to route userrequests and communications to preferred or particular other agents,human or virtual.

The human agents 60 may utilize agent devices 62 to serve users in theirinteractions to communicate and take action. The agent devices 62 canbe, as non-limiting examples, computing devices, kiosks, terminals,smart devices such as phones, and devices and tools at customer servicecounters and windows at POS locations. In at least one example, thediagrammatic representation of the components of the mobile device 16 inFIG. 1 applies as well to one or both of the computing device 14 and theagent devices 62.

The agent devices 62 individually or collectively include input devicesand output devices, including, as non-limiting examples, a touch screen,which serves both as an output device by providing graphical and textindicia and presentations for viewing by one or more of the agents 60,and as an input device by providing virtual buttons, selectable options,a virtual keyboard, and other indicia that, when touched or activated,control or prompt the agent device 62 by action of the attendant agent60. Further non-limiting examples include, one or more of each, any, andall of a keyboard, a mouse, a touchpad, a joystick, a button, a switch,a light, an LED, a microphone serving as input device for example forvoice input by the human agent 60, a speaker serving as an outputdevice, a camera serving as an input device, a buzzer, a bell, a printerand/or other user input devices and output devices for use by orcommunication with the human agent 60 in accessing, using, andcontrolling, in whole or in part, the agent device 62.

Inputs by one or more of the human agents 60 can thus be made via voice,text or graphical indicia selections. For example, some inputs receivedby the agent device 62 in some examples correspond to, control, orprompt enterprise-side actions and communications offering services andproducts of the enterprise system 12, information thereof, or accessthereto. At least some outputs by the agent device 62 in some examplescorrespond to, or are prompted by, user-side actions and communicationsin two-way communications between the user 18 and an enterprise-sidehuman agent 60.

From a user perspective experience, an interaction in some exampleswithin the scope of these descriptions begins with direct or firstaccess to one or more of the human agents 60 in person, by phone oronline for example via a chat session or website function or feature. Inother examples, a user is first assisted by a virtual agent 64 of theenterprise system 12, which may satisfy user requests or prompts byvoice, text or online functions, and may refer users to one or more ofthe human agents 60 once preliminary determinations or conditions aremade or met.

The enterprise system 12 includes a computing system 70 having variouscomponents, such as a processing device 72 and a memory device 74 forprocessing use, such as random access memory (RAM) and read-only memory(ROM). The computing system 70 further includes a storage device 76having at least one non-transitory storage medium, such as a microdrive,for long-term, intermediate-term, and short-term storage ofcomputer-readable instructions 78 for execution by the processing device72. For example, the instructions 78 can include instructions for anoperating system and various applications or programs 80, of which anapplication 82 is represented as a particular example. The storagedevice 76 can store various other data 84, which can include, asnon-limiting examples, cached data, and files such as those for useraccounts, user profiles, account balances, and transaction histories,files downloaded or received from other devices, and other data itemspreferred by the user or required or related to any or all of theapplications or programs 80.

The computing system 70, in the illustrated example, also includes aninput/output system 86, referring to, including, or operatively coupledwith input devices and output devices such as, in a non-limitingexample, agent devices 62, which have both input and outputcapabilities.

In the illustrated example, a system intraconnect 88 electricallyconnects the various above-described components of the computing system70. In some cases, the intraconnect 88 operatively couples components toone another, which indicates that the components may be directly orindirectly connected, such as by way of one or more intermediatecomponents. The intraconnect 88, in various non-limiting examples, caninclude or represent, a system bus, a high-speed interface connectingthe processing device 72 to the memory device 74, individual electricalconnections among the components, and electrical conductive traces on amotherboard common to some or all of the above-described components ofthe user device.

The computing system 70 includes a communication interface 90 by whichthe computing system 70 communicates and conducts transactions withother devices and systems. The communication interface 90 may includedigital signal processing circuitry and may provide two-waycommunications and data exchanges, for example wirelessly via wirelessdevice 92, and for an additional or alternative example, via wired ordocked communication by mechanical electrically conductive connector 94.Communications may be conducted via various modes or protocols, of whichGSM voice calls, SMS, EMS, MMS messaging, TDMA, CDMA, PDC, WCDMA,CDMA2000, and GPRS, are all non-limiting and non-exclusive examples.Thus, communications can be conducted, for example, via the wirelessdevice 92, which can be or include a radio-frequency transceiver, aBluetooth device, Wi-Fi device, near-field communication device, andother transceivers. In addition, GPS may be included for navigation andlocation-related data exchanges, ingoing and/or outgoing. Communicationsmay also or alternatively be conducted via the connector 94 for wiredconnections such as by USB, Ethernet, and other physically connectedmodes of data transfer.

The processing device 72, in various examples, can operatively performcalculations, can process instructions for execution, and can manipulateinformation. The processing device 72 can execute machine-executableinstructions stored in the storage device 76 and/or the memory device 74to thereby perform methods and functions as described or implied herein,for example by one or more corresponding flow charts expressly providedor implied as would be understood by one of ordinary skill in the art towhich the subjects matters of these descriptions pertain. The processingdevice 72 can be or can include, as non-limiting examples, a centralprocessing unit (CPU), a microprocessor, a graphics processing unit(GPU), a microcontroller, an application-specific integrated circuit(ASIC), a programmable logic device (PLD), a digital signal processor(DSP), a field programmable gate array (FPGA), a state machine, acontroller, gated or transistor logic, discrete physical hardwarecomponents, and combinations thereof.

Furthermore, the computing system 70, may be or include a workstation, aserver, or any other suitable device, including a set of servers, acloud-based application or system, or any other suitable system, adaptedto execute, for example any suitable operating system, including Linux,UNIX, Windows, macOS, iOS, Android, and any known other operating systemused on personal computer, central computing systems, phones, and otherdevices.

The user devices, referring to either or both of the mobile device 16and the computing device 14, the agent devices 62 and the computingsystem 70, which may be one or any number centrally located ordistributed, are in communication through one or more networks,referenced as system 10 in FIG. 1 .

The network 100 provides wireless or wired communications among thecomponents of the network 100 and the environment thereof, includingother devices local or remote to those illustrated, such as additionalmobile devices, servers, and other devices communicatively coupled tothe network 100, including those not illustrated in FIG. 1 . The network100 is singly depicted for illustrative convenience, but may includemore than one network without departing from the scope of thesedescriptions. In some embodiments, the network 100 may be or provide oneor more cloud-based services or operations. The network 100 may be orinclude an enterprise or secured network, or may be implemented, atleast in part, through one or more connections to the Internet. Aportion of the network 100 may be a virtual private network (VPN) or anIntranet. The network 100 can include wired and wireless links,including, as non-limiting examples, 802.11a/b/g/n/ac, 802.20, WiMax,LTE, and/or any other wireless link. The network 100 may include anyinternal or external network, networks, sub-network, and combinations ofsuch operable to implement communications between various computingcomponents within and beyond the network 100. The network 100 maycommunicate, for example, internet protocol (IP) packets, frame relayframes, asynchronous transfer mode (ATM) cells, voice, video, data, andother suitable information between network addresses. The network 100may also include one or more local area networks (LANs), radio accessnetworks (RANs), metropolitan area networks (MANs), wide area networks(WANs), all or a portion of the internet and/or any other communicationsystem or systems at one or more locations.

Two external systems 102 and 104 are illustrated in FIG. 1 andrepresenting any number and variety of data sources, users, consumers,customers, business entities, banking systems, government entities,clubs, and groups of any size are all within the scope of thedescriptions. In at least one example, the external systems 102 and 104represent automatic teller machines (ATMs) utilized by the enterprisesystem 12 in serving the users 18. In another example, the externalsystems 102 and 104 represent payment clearinghouse or payment railsystems for processing payment transactions, and in another example, theexternal systems 102 and 104 represent third party systems such asmerchant systems configured to interact with the user device 16 duringtransactions and also configured to interact with the enterprise system12 in back-end transactions clearing processes.

In certain embodiments, one or more of the systems such as the userdevice 16, the enterprise system 12, and/or the external systems 102 and104 are, include, or utilize virtual resources. In some cases, suchvirtual resources are considered cloud resources or virtual machines.Such virtual resources may be available for shared use among multipledistinct resource consumers and in certain implementations, virtualresources do not necessarily correspond to one or more specific piecesof hardware, but rather to a collection of pieces of hardwareoperatively coupled within a cloud computing configuration so that theresources may be shared as needed.

As used herein, an artificial intelligence system, artificialintelligence algorithm, artificial intelligence module, program, and thelike, generally refer to computer implemented programs that are suitableto simulate intelligent behavior (i.e., intelligent human behavior)and/or computer systems and associated programs suitable to performtasks that typically require a human to perform, such as tasks requiringvisual perception, speech recognition, decision-making, translation, andthe like. An artificial intelligence system may include, for example, atleast one of a series of associated if-then logic statements, astatistical model suitable to map raw sensory data into symboliccategories and the like, or a machine learning program. A machinelearning program, machine learning algorithm, or machine learningmodule, as used herein, is generally a type of artificial intelligenceincluding one or more algorithms that can learn and/or adjust parametersbased on input data provided to the algorithm. In some instances,machine learning programs, algorithms and modules are used at least inpart in implementing artificial intelligence (AI) functions, systems andmethods.

Artificial Intelligence and/or machine learning programs may beassociated with or conducted by one or more processors, memory devices,and/or storage devices of a computing system or device. It should beappreciated that the artificial intelligence algorithm or program may beincorporated within the existing system architecture or be configured asa standalone modular component, controller, or the like communicativelycoupled to the system. An artificial intelligence program and/or machinelearning program may generally be configured to perform methods andfunctions as described or implied herein, for example by one or morecorresponding flow charts expressly provided or implied as would beunderstood by one of ordinary skill in the art to which the subjectsmatters of these descriptions pertain.

A machine learning program may be configured to implement storedprocessing, such as decision tree learning, association rule learning,artificial neural networks, recurrent artificial neural networks, longshort term memory networks, inductive logic programming, support vectormachines, clustering, Bayesian networks, reinforcement learning,representation learning, similarity and metric learning, sparsedictionary learning, genetic algorithms, k-nearest neighbor (KNN), andthe like. In some embodiments, the machine learning algorithm mayinclude one or more image recognition algorithms suitable to determineone or more categories to which an input, such as data communicated froma visual sensor or a file in JPEG, PNG or other format, representing animage or portion thereof, belongs. Additionally or alternatively, themachine learning algorithm may include one or more regression algorithmsconfigured to output a numerical value given an input. Further, themachine learning may include one or more pattern recognition algorithms,e.g., a module, subroutine or the like capable of translating text orstring characters and/or a speech recognition module or subroutine. Invarious embodiments, the machine learning module may include a machinelearning acceleration logic, e.g., a fixed function matrixmultiplication logic, in order to implement the stored processes and/oroptimize the machine learning logic training and interface.

One type of algorithm suitable for use in machine learning modules asdescribed herein is an artificial neural network or neural network,taking inspiration from biological neural networks. An artificial neuralnetwork can learn to perform tasks by processing examples, without beingprogrammed with any task-specific rules. A neural network generallyincludes connected units, neurons, or nodes (e.g., connected bysynapses) and may allow for the machine learning program to improveperformance. A neural network may define a network of functions, whichhave a graphical relationship. As an example, a feedforward network maybe utilized, such as an acyclic graph with nodes arranged in layers.

The artificial intelligence systems and structures discussed herein mayemploy deep learning. Deep learning is a particular type of machinelearning that provides greater learning performance by representing acertain real-world environment as a hierarchy of increasing complexconcepts. Deep learning typically employs a software structurecomprising several layers of neural networks that perform nonlinearprocessing, where each successive layer receives an output from theprevious layer. Generally, the layers include an input layer thatreceives raw data from a sensor, a number of hidden layers that extractabstract features from the data, and an output layer that identifies acertain thing based on the feature extraction from the hidden layers.The neural networks include neurons or nodes that each has a “weight”that is multiplied by the input to the node to obtain a probability ofwhether something is correct. More specifically, each of the nodes has aweight that is a floating point number that is multiplied with the inputto the node to generate an output for that node that is some proportionof the input. The weights are initially “trained” or set by causing theneural networks to analyze a set of known data under supervisedprocessing and through minimizing a cost function to allow the networkto obtain the highest probability of a correct output. Deep learningneural networks are often employed to provide image feature extractionand transformation for the visual detection and classification ofobjects in an image, where a video or stream of images can be analyzedby the network to identify and classify objects and learn through theprocess to better recognize the objects. Thus, in these types ofnetworks, the system can use the same processing configuration to detectcertain objects and classify them differently based on how the algorithmhas learned to recognize the objects.

FIG. 2 illustrates a feedforward neural network 110 that includes ahidden layer 114 between an input layer 112 and an output layer 116. Theinput layer 112, having nodes commonly referenced in FIG. 2 as inputnodes 118 for convenience, communicates input data, variables, matrices,or the like to the hidden layer 114, having nodes 120. The hidden layer114 generates a representation and/or transformation of the input datainto a form that is suitable for generating output data. Adjacent layersof the neural network 110 are connected at the edges of the nodes of therespective layers, but nodes within a layer typically are not separatedby an edge. In at least one embodiment of such a feedforward neuralnetwork, data is communicated to the nodes 118 of the input layer 112,which then communicates the data to the hidden layer 114. The hiddenlayer 114 may be configured to determine the state of the nodes in therespective layers and assign weight coefficients or parameters of thenodes based on the edges separating each of the layers, such as anactivation function implemented between the input data communicated fromthe input layer 112 and the output data communicated to nodes 122 of theoutput layer 116. It should be appreciated that the form of the outputfrom the neural network may generally depend on the type of modelrepresented by the algorithm. Although the feedforward neural network110 expressly includes a single hidden layer, other embodiments offeedforward networks within the scope of the descriptions can includeany number of hidden layers. The hidden layers are intermediate theinput and output layers and are generally where all or most of thecomputation is done.

Neural networks may perform a supervised learning process where knowninputs and known outputs are utilized to categorize, classify, orpredict a quality of a future input. However, additional or alternativeembodiments of the machine learning program may be trained utilizingunsupervised or semi-supervised training, where none of the outputs orsome of the outputs are unknown, respectively. Typically, a machinelearning algorithm is trained, for example, utilizing a training dataset, prior to modeling the problem with which the algorithm isassociated. Supervised training of the neural network may includechoosing a network topology suitable for the problem being modeled bythe network and providing a set of training data representative of theproblem. Generally, the machine learning algorithm may adjust the weightcoefficients until any error in the output data generated by thealgorithm is less than a predetermined, acceptable level. For instance,the training process may include comparing the generated output producedby the network in response to the training data with a desired orcorrect output. An associated error amount may then be determined forthe generated output data, such as for each output data point generatedin the output layer. The associated error amount may be communicatedback through the system as an error signal, where the weightcoefficients assigned in the hidden layer are adjusted based on theerror signal. For instance, the associated error amount, such as a valuebetween−1 and 1, may be used to modify the previous coefficient, e.g., apropagated value. The machine learning algorithm may be consideredsufficiently trained when the associated error amount for the outputdata is less than the predetermined, acceptable level, for example, eachdata point within the output layer includes an error amount less thanthe predetermined, acceptable level. Thus, the parameters determinedfrom the training process can be utilized with new input data tocategorize, classify, and/or predict other values based on the new inputdata.

An additional or alternative type of neural network suitable for use ina machine learning program and/or module is a convolutional neuralnetwork (CNN). A CNN is a type of feedforward neural network that may beutilized to model data associated with input data having a grid-liketopology. In some embodiments, at least one layer of a CNN may include asparsely connected layer, in which each output of a first hidden layerdoes not interact with each input of the next hidden layer. For example,the output of the convolution in the first hidden layer may be an inputof the next hidden layer, rather than a respective state of each node ofthe first layer. CNNs are typically trained for pattern recognition,such as speech processing, language processing, and visual processing.As such, CNNs may be particularly useful for implementing optical andpattern recognition programs required from the machine learning program.A CNN includes an input layer, a hidden layer, and an output layer,typical of feedforward networks, but the nodes of a CNN input layer aregenerally organized into a set of categories via feature detectors andbased on the receptive fields of the sensor, retina, input layer, etc.Each filter may then output data from its respective nodes tocorresponding nodes of a subsequent layer of the network. A CNN may beconfigured to apply the convolution mathematical operation to therespective nodes of each filter and communicate the same to thecorresponding node of the next subsequent layer. As an example, theinput to the convolution layer may be a multidimensional array of data.The convolution layer, or hidden layer, may be a multidimensional arrayof parameters determined while training the model.

FIG. 3 is an illustration of an exemplary CNN 130 that includes an inputlayer 132 and an output layer 134. However, where the single hiddenlayer 114 is provided in the network 110, multiple consecutive hiddenlayers 136, 138 and 140 are provided in the CNN 130. Edge neurons 142represented by white-filled arrows highlight that hidden layer nodes 144can be connected locally, such that not all of the nodes of succeedinglayers are connected by neurons.

FIG. 4 shows a portion of the CNN 130, specifically portions of theinput layer 132 and the first hidden layer 136, and illustrates thatconnections can be weighted. In the illustrated example, labels W1 andW2 refer to respective assigned weights for the referenced connections.The two hidden nodes 146 and 148 share the same set of weights W1 and W2when connecting to two local patches.

A weight defines the impact a node in any given layer has oncomputations by a connected node in the next layer. FIG. 5 shows anetwork 150 including a node 152 in a hidden layer. The node 152 isconnected to several nodes in the previous layer representing inputs tothe node 152. Input nodes 154, 156, 158 and 160 in an input layer 162are each assigned a respective weight W01, W02, W03, and W04 in thecomputation at the node 152, which in this example is a weighted sum.

An additional or alternative type of feedforward neural network suitablefor use in the machine learning program and/or module is a recurrentneural network (RNN). An RNN may allow for analysis of sequences ofinputs rather than only considering the current input data set. RNNstypically include feedback loops/connections between layers of thetopography, thus allowing parameter data to be communicated betweendifferent parts of the neural network. RNNs typically have anarchitecture including cycles, where past values of a parameterinfluence the current calculation of the parameter, e.g., at least aportion of the output data from the RNN may be used as feedback/input incalculating subsequent output data. In some embodiments, the machinelearning module may include an RNN configured for language processing,e.g., an RNN configured to perform statistical language modeling topredict the next word in a string based on the previous words. TheRNN(s) of the machine learning program may include a feedback systemsuitable to provide the connection(s) between subsequent and previouslayers of the network.

FIG. 6 illustrates an RNN 170 that includes an input layer 172 withnodes 174, an output layer 176 with nodes 178, and multiple consecutivehidden layers 180 and 182 with nodes 184 and nodes 186, respectively.The RNN 170 also includes a feedback connector 188 configured tocommunicate parameter data from at least one of the nodes 186 in thesecond hidden layer 184 to at least one of the nodes 182 in the firsthidden layer 184. It should be appreciated that two or more and up toall of the nodes of a subsequent layer may provide or communicate aparameter or other data to a previous layer of the RNN 170. Moreover andin some embodiments, the RNN 170 may include multiple feedbackconnectors, such as connectors suitable to communicatively couple pairsof nodes and/or connector systems configured to provide communicationbetween three or more nodes. Additionally or alternatively, the feedbackconnector 188 may communicatively couple two or more nodes having atleast one hidden layer between them, i.e., nodes of non-sequentiallayers of the RNN 170.

In an additional or alternative embodiment, the machine learning programmay include one or more support vector machines. A support vectormachine may be configured to determine a category to which input databelongs. For example, the machine learning program may be configured todefine a margin using a combination of two or more of the inputvariables and/or data points as support vectors to maximize thedetermined margin. Such a margin may generally correspond to a distancebetween the closest vectors that are classified differently. The machinelearning program may be configured to utilize a plurality of supportvector machines to perform a single classification. For example, themachine learning program may determine the category to which input databelongs using a first support vector determined from first and seconddata points/variables, and the machine learning program mayindependently categorize the input data using a second support vectordetermined from third and fourth data points/variables. The supportvector machine(s) may be trained similarly to the training of neuralnetworks, e.g., by providing a known input vector (including values forthe input variables) and a known output classification. The supportvector machine is trained by selecting the support vectors and/or aportion of the input vectors that maximize the determined margin.

As depicted, and in some embodiments, the machine learning program mayinclude a neural network topography having more than one hidden layer.In such embodiments, one or more of the hidden layers may have adifferent number of nodes and/or the connections defined between layers.In some embodiments, each hidden layer may be configured to perform adifferent function. As an example, a first layer of the neural networkmay be configured to reduce a dimensionality of the input data, and asecond layer of the neural network may be configured to performstatistical programs on the data communicated from the first layer. Invarious embodiments, each node of the previous layer of the network maybe connected to an associated node of the subsequent layer (denselayers). Generally, the neural network(s) of the machine learningprogram may include a relatively large number of layers, e.g., three ormore layers, and are referred to as deep neural networks. For example,the node of each hidden layer of a neural network may be associated withan activation function utilized by the machine learning program togenerate an output received by a corresponding node in the subsequentlayer. The last hidden layer of the neural network communicates a dataset (e.g., the result of data processed within the respective layer) tothe output layer. Deep neural networks may require more computationaltime and power to train, but the additional hidden layers providemultistep pattern recognition capability and/or reduced output errorrelative to simple or shallow machine learning architectures (e.g.,including only one or two hidden layers).

FIG. 7 is a block diagram of an artificial intelligence programmingsystem 200 including an AI processor 202, such as a dedicated processingdevice, that operates an artificial intelligence program, where theprocessor 202 includes a front-end sub-processor 204 and a back-endsub-processor 206. The algorithms associated with the front-endsub-processor 204 and the back-end sub-processor 206 may be stored in anassociated memory device and/or storage device, such as memory device208 communicatively coupled to the AI processor 202, as shown.Additionally, the system 200 may include a memory 212 storing one ormore instructions necessary for operating the AI program. In thisembodiment, the sub-processor 204 includes neural networks 214 and 216operating an AI algorithm 218, such as feature recognition, and thesub-processor 206 includes neural networks 220 and 222 operating an AIalgorithm 224 to perform an operation on the data set communicateddirectly or indirectly to the sub-processor 206.

The system 200 may provide statistical models or machine learningprograms such as decision tree learning, associate rule learning,recurrent artificial neural networks, support vector machines, and thelike. In various embodiments, the sub-processor 204 may be configured toinclude built in training and inference logic or suitable software totrain the neural network prior to use, for example, machine learninglogic including, but not limited to, image recognition, mapping andlocalization, autonomous navigation, speech synthesis, document imaging,or language translation. For example, the sub-processor 204 may be usedfor image recognition, input categorization, and/or support vectortraining. In various embodiments, the sub-processor 206 may beconfigured to implement input and/or model classification, speechrecognition, translation, and the like.

For instance and in some embodiments, the system 200 may be configuredto perform unsupervised learning, in which the machine learning programperforms the training process using unlabeled data, e.g., without knownoutput data with which to compare. During such unsupervised learning,the neural network may be configured to generate groupings of the inputdata and/or determine how individual input data points are related tothe complete input data set. For example, unsupervised training may beused to configure a neural network to generate a self-organizing map,reduce the dimensionally of the input data set, and/or to performoutlier/anomaly determinations to identify data points in the data setthat falls outside the normal pattern of the data. In some embodiments,the system 200 may be trained using a semi-supervised learning processin which some but not all of the output data is known, e.g., a mix oflabeled and unlabeled data having the same distribution.

In some embodiments, the system 200 may include an index of basicoperations, subroutines, and the like (primitives) typically implementedby AI and/or machine learning algorithms. Thus, the system 200 may beconfigured to utilize the primitives of the processor 202 to performsome or all of the calculations required by the system 200. Primitivessuitable for inclusion in the processor 202 include operationsassociated with training a convolutional neural network (e.g., pools),tensor convolutions, activation functions, basic algebraic subroutinesand programs (e.g., matrix operations, vector operations), numericalmethod subroutines and programs, and the like.

It should be appreciated that the machine learning program may includevariations, adaptations, and alternatives suitable to perform theoperations necessary for the system, and the present disclosure isequally applicable to such suitably configured machine learning and/orartificial intelligence programs, modules, etc. For instance, themachine learning program may include one or more long short-term memory(LSTM) RNNs, convolutional deep belief networks, deep belief networksDBNs, and the like. DBNs, for instance, may be utilized to pre-train theweighted characteristics and/or parameters using an unsupervisedlearning process. Further, the machine learning module may include oneor more other machine learning tools (e.g., logistic regression (LR),Naive-Bayes, random forest (RF), matrix factorization, and supportvector machines) in addition to, or as an alternative to, one or moreneural networks, as described herein.

FIG. 8 is a flow chart diagram 230 showing an exemplary method for modeldevelopment and deployment by machine learning. The method represents atleast one example of a machine learning workflow in which steps areimplemented in a machine learning project. At box 232, a userauthorizes, requests, manages, or initiates the machine-learningworkflow. This may represent a user such as human agent, or customer,requesting machine-learning assistance or AI functionality to simulateintelligent behavior, such as a virtual agent, or other machine-assistedor computerized tasks that may, for example, entail visual perception,speech recognition, decision-making, translation, forecasting,predictive modelling, and/or suggestions as non-limiting examples. In afirst iteration from the user perspective, the box 232 can represent astarting point. However, with regard to continuing or improving anongoing machine learning workflow, the box 232 can represent anopportunity for further user input or oversight via a feedback loop.

At box 234, data is received, collected, accessed or otherwise acquiredand entered as can be termed data ingestion. At box 236, data ingestedfrom the box 234 is pre-processed, for example, by cleaning, and/ortransformation such as into a format that the following components candigest. The incoming data may be versioned to connect a data snapshotwith the particularly resulting trained model. As newly trained modelsare tied to a set of versioned data, preprocessing steps are tied to thedeveloped model. If new data is subsequently collected and entered, anew model will be generated. If the preprocessing is updated with newlyingested data, an updated model will be generated. The process at thebox 236 can include data validation, which focuses on confirming thatthe statistics of the ingested data are as expected, such as that datavalues are within expected numerical ranges, that data sets are withinany expected or required categories, and that data comply with anyneeded distributions such as within those categories. The process canproceed to box 238 to automatically alert the initiating user, otherhuman or virtual agents, and/or other systems, if any anomalies aredetected in the data, thereby pausing or terminating the process flowuntil corrective action is taken.

At box 240, training test data, such as a target variable value, isinserted into an iterative training and testing loop. At box 242, modeltraining, a core step of the machine learning work flow, is implemented.A model architecture is trained in the iterative training and testingloop. For example, features in the training test data are used to trainthe model based on weights and iterative calculations in which thetarget variable may be incorrectly predicted in an early iteration asdetermined by comparison at box 244, where the model is tested.Subsequent iterations of the model training at the box 242 may beconducted with updated weights in the calculations.

When compliance and/or success in the model testing at the box 244 isachieved, the process proceeds to box 246, where model deployment istriggered. The model may be utilized in AI functions and programming,for example, to simulate intelligent behavior, to performmachine-assisted or computerized tasks, of which visual perception,speech recognition, decision-making, translation, forecasting,predictive modelling, and/or automated suggestion generation serve asnon-limiting examples.

FIG. 9 is a flow diagram 250 illustrating a process for identifyinghandwritten characters on an image that has application forautomatically identifying the payee on an image of a handwritten checkwhen it is processed by a bank. A scanned image 252 of a check beingprocessed is received by the bank from, for example, the ACH. The image252 of the check includes a signature line 254, a payee line 256, apayor line 258 and a date line 260. The image 252 is subjected to anoptical character recognition (OCR) algorithm at box 262 that reads andconverts typed and/or printed text on the image 252 into machine encodedtext in a manner well understood by those skilled in the art. Onesuitable OCR algorithm for this purpose is known as Tesseract™, which isa Python™ algorithm, although others may be equally applicable. Thesetypes of known OCR algorithms are able to read and convert typed text inthe image 252 with a high degree of accuracy or confidence, but can onlyread and convert handwritten text with a low and unacceptable accuracyor confidence. One of those typed text portions on a conventional checkis the words “pay to the order of” positioned next to the payee line 256of the check. The OCR algorithm is programmed to identify the locationof the words “pay to the order of”, or some other suitable typed text inthe image 252, and then using that location to provide the x and ycoordinates of the pixels in the image 252 for the location of the payeeline 256 in the image 252 so that the area of the image 252 having thehandwritten payee can be further processed using a more accuratecharacter recognition process to identify the payee, as will bediscussed below. The typed text identified in other areas of the image252, such as the routing number, account number, etc. can also beprocessed for other purposes, if desired, such as for identifying fraudissues.

The image 252 is then subjected to a density-based clustering algorithmthat provides character segmentation at box 264 that also receives thelocation of the payee line 256 in the image 252. The clusteringalgorithm identifies clusters of non-white pixels in the area of theimage 252 identified by the OCR algorithm, where each separatelyidentified cluster may be a handwritten character that is part of thepayee handwritten in the payee line 256. Particularly, the section ofthe image 252 being looked at is processed as a matrix of pixels, whereeach non-white pixel is considered a data point for the clusteringprocess. One suitable density-based clustering algorithm is DBScan™,which is a density-based clustering non-parametric algorithm. Given aset of points in some space, the DBScan™ algorithm groups togetherpoints that are closely packed together, i.e., points with many nearbyneighbors, marking as outliers points that lie alone in low-densityregions whose nearest neighbors are too far away. The output of theclustering algorithm is a dataset array that digitally identifies the xand y coordinates of the pixels in each identified character clusteralong with an assigned label for each cluster, where the algorithm willassign the same cluster label to data points that are part of the samecluster.

The clustering algorithm is effective for identifying clusters of pixelsin the image 252 that are part of the same character. However, someletters, such as, for example, capital I, may look like two clusters tothe algorithm because of the top and bottom bars in the letter.Handwriting heuristics can be employed to add constraints to theclustering algorithm to reduce the chance that more than one clusterdataset array is identified for the same character and/or one clusterdataset includes more than one character. For example, since the payeewill be written from left to right on the check, the clusteringalgorithm could be designed with a constraint where clusters cannot bestacked top to bottom on the check relative to the handwriting directionof the payee. In other words, if two clusters are identified in avertical direction at the same left to right location on the image 252,then that cluster can be considered a single cluster for one character.Further, a limit to the size or area of the cluster can also be aconstraint. For example, if the size of the cluster exceeds somethreshold, then it is assumed that the cluster includes more than onecharacter and the clustering of that cluster can be further defined andprocessed to identify the multiple clusters. In other words, aconstraint can be employed that limits the size of each cluster to beless than a predetermined maximum size. Also, a person will typicallywrite in a manner where all of the characters are about the same size orwidth. If there is a significant inconsistency in the size of theclusters, then the cluster process can be further refined to identifyadditional clusters. In other words, a constraint can be employed thatrequires all of the clusters to be within a certain percentage size ofeach other.

The dataset array from the clustering algorithm is provided to a clusterextraction and rescaling algorithm at box 266 that extracts theindividually identified clusters in the dataset array into individualdataset arrays and rescales each individual dataset array into, forexample, a 28×28 pixel cluster image, using extrapolation, which keepsthe main features of the image 252. The rescaling process also centersthe cluster in the cluster image and adds border padding. FIG. 10 is arepresentation of a 28×28 pixel cluster image 274 illustrating anidentified cluster for a handwritten letter O after having beenprocessed by the cluster extraction and rescaling algorithm. The sizeand format of the image 274 is selected to be compatible with acharacter classification process discussed below.

Each individual cluster image 274 is provided to a characterclassification model at box 268 that classifies the likelihood that eachcluster image 274 is a particular character. The classification modelemploys a neural network, specifically a CNN, such as neural network 276shown in FIG. 11 . The neural network 276 includes an input layer 278that receives the cluster image 274, a convolutional layer 280 thatclassifies the image 274, a pooling layer 282 that reduces thedimensions of feature maps, a fully connected layer 284 that connectsthe nodes between layers, and an output layer 286 that outputs theclassified characters. The neural network 276 is trained using a knownset of training images, where each training image illustrates a letteror a number that has been assigned one of sixty-two character classes,namely, the upper case letters A-Z, the lower case letters a-z and thenumbers 0-9. More particularly, nodes in the neural network 276 areweighted and those weights are tuned during the training process toallow the neural network 276 to determine what locations in the image274 that include non-white pixels are what character. The trainingprocess first feeds the neural network 276 character training images andtells the neural network 276 what those characters are, and then feedsthe neural network 276 character training images without telling theneural network 276 what the characters are, where the weights areadjusted based on right character identification answers and wrongcharacter identification answers. The known set of training images couldbe the EMNIST™ dataset, which is a set of 697,932 handwritten characterdigits derived from the NIST Special Database 19 and converted to a28×28 pixel image format and dataset structure.

During operation, the neural network 276 determines the likelihood thateach image 274 is each of the sixty-two characters, and selects thecharacter with the highest likelihood as the character for that image274. The classification model outputs a string of characters from leftto right along the payee line 256, in this example, and identifiesspaces between certain characters where non-white pixels do not exist asan indication of a separation between words. The classification modelcan employ any algorithm suitable for the purposes described herein,such as Pytorch™, which is an open source machine learning frameworkbased on the Torch library, used for applications such as computervision and natural language processing.

The image 252, the typed text on the image 252 from the OCR and thecharacter classification and location are sent to a payee identificationprocess at box 270 where the payee is identified to be available to beuploaded to a user's account or transaction register. The classificationof the characters may not be 100% accurate, so further operations can beprovided to help identify the payee in the event that some of thecharacters are misidentified. For example, typed zip code and addressinformation on the image 252 can be obtained from the OCR to determine ageographic location of the payee. The payee at that location can besearched using a secondary data source, such as Google Search API™, orsome type of autocorrecting function, at box 272 to specificallyidentify the payee based on the characters that were identified. Forexample, the identified payee can be automatically searched on theinternet to see if it exists, and if so, where.

Particular embodiments and features have been described with referenceto the drawings. It is to be understood that these descriptions are notlimited to any single embodiment or any particular set of features.Similar embodiments and features may arise or modifications andadditions may be made without departing from the scope of thesedescriptions and the spirit of the appended claims.

What is claimed is:
 1. A system for identifying handwritten characterson an image, said system comprising: a computer including at least oneprocessor and a memory device storing data and executable code that,when executed, causes the at least one processor to: read and converttyped text on the image to machine encoded text to identify locations ofthe typed text on the image; identify a location on the image thatincludes handwritten text based on the location of predetermined typedtext on the image; identify clusters of non-white pixels in the image atthe location having the handwritten text, where each cluster is presumedto be a handwritten character, wherein the at least one processoremploys a constraint that limits a size of each cluster to be less thana predetermined size; generate an individual and separate cluster imagefor each identified cluster; classify each cluster image using machinelearning and at least one neural network to determine the likelihoodthat the cluster is a certain character; and determine what charactereach cluster image is based on the classification.
 2. The systemaccording to claim 1 wherein the at least one processor reads andconverts the typed text using an optical character recognitionalgorithm.
 3. The system according to claim 1 wherein the at least oneprocessor identifies clusters of non-white pixels using a density-basedclustering algorithm.
 4. The system according to claim 1 wherein the atleast one processor rescales the cluster when it generates an individualand separate cluster image for each identified cluster.
 5. The systemaccording to claim 4 wherein the at least one processor rescales thecluster to a 28×28 pixel cluster image.
 6. The system according to claim1 wherein the at least one processor centers the cluster in the clusterimage when it generates an individual and separate cluster image foreach identified cluster.
 7. The system according to claim 1 wherein theat least one processor classifies each cluster image by determining thelikelihood that the cluster image is one of sixty-two character classes,namely, upper case letters A-Z, lower case letters a-z and numbers 0-9.8. The system according to claim 1 wherein the at least one neuralnetwork is a convolutional neural network (CNN).
 9. A system foridentifying handwritten characters on an image of a check, said systemcomprising: a computer including at least one processor and a memorydevice storing data and executable code that, when executed, causes theat least one processor to: read and convert typed text on the image tomachine encoded text using an optical character recognition algorithm toidentify locations of the typed text on the image including identifyingthe location of the typed words “pay to the order of”; identify alocation on the image that includes a handwritten payee based on thelocation of the typed words “pay to the order of”; identify clusters ofnon-white pixels in the image at the location having the handwrittenpayee using a density-based clustering algorithm, where each cluster ispresumed to be a handwritten character, wherein the at least oneprocessor employs a constraint that limits a size of each cluster to beless than a predetermined size; generate an individual and separatecluster image for each identified cluster; classify each cluster imageusing machine learning and at least one neural network to determine thelikelihood that the cluster is a certain character, wherein the at leastone processor classifies each cluster image by determining thelikelihood that the cluster image is one of sixty-two character classes,namely, upper case letters A-Z, lower case letters a-z and numbers 0-9;and determine what character each cluster image is based on theclassification.
 10. The system according to claim 9 wherein the at leastone processor rescales the cluster when it generates an individual andseparate cluster image for each identified cluster.
 11. The systemaccording to claim 10 wherein the at least one processor rescales thecluster to a 28×28 pixel cluster image.
 12. The system according toclaim 9 wherein the at least one processor centers the cluster in thecluster image when it generates an individual and separate cluster imagefor each identified cluster.
 13. A method for identifying handwrittencharacters on an image, said method comprising: reading and convertingtyped text on the image to machine encoded text to identify locations ofthe typed text on the image; identifying a location on the image thatincludes handwritten text based on the location of predetermined typedtext on the image; identifying clusters of non-white pixels in the imageat the location having the handwritten text, where each cluster ispresumed to be a handwritten character, wherein identifying clusters ofnon-white pixels includes employing that limits a size of each clusterto be less than a predetermined size; generating an individual andseparate cluster image for each identified cluster, classifying eachcluster image using machine learning and at least one neural network todetermine the likelihood that the cluster is a certain character; anddetermining what character each cluster image is based on theclassification.
 14. The method according to claim 13 wherein reading andconverting the typed text includes using an optical characterrecognition algorithm and identifying clusters of non-white pixelsincludes using a density-based clustering algorithm.
 15. The methodaccording to claim 13 wherein generating an individual and separatecluster image for each identified cluster includes rescaling eachcluster.
 16. The method according to claim 15 wherein each cluster isrescaled to a 28×28 pixel cluster image.
 17. The method according toclaim 13 wherein generating an individual and separate cluster image foreach identified cluster includes centering the cluster in the image. 18.The method according to claim 13 wherein the at least one processorclassifies each cluster image by determining the likelihood that thecluster image is one of sixty-two character classes, namely, upper caseletters A-Z, lower case letters a-z and numbers 0-9.
 19. The methodaccording to claim 13 wherein the at least one neural network is aconvolutional neural network (CNN).
 20. The method according to claim 13wherein the image is an image of a check, the predetermined typed textis “pay to the order of” and the handwritten text is a payee on thecheck.